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AESTBACT 

This paper discusses the use of process-oriented 
research in early childhood programs. Process analysis is contrasted 
with outcome-oriented evaluation, the latter seen as prevalent ::n 
current research practice. Four functions of process analysis are: to 
describe current teaching practices, to train teachers, to monitor 
instructional programs, and to investigate relationships betueen 
classroom activities and pupil growth. Systems for observing the 
process of teacher-child interaction are categorized as either open, 
low-inference systems or closed, high-inference systems. The 
specimen-description technique is cited as an example of the forn^r, 
while examples of closed systems are time-sampling, event-sampling 
and txa.it-rating. Methodological issues dif,cussed relate to choice of 
approp\:iate instruments, reliability of observers, and degree of 
researcher control over selection and supervision of the program to 
be evaluated. (Authcr/BF) 
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EVALUATION QP PRESQIOOL PROGRAMS 



The pifcliferation of day care facilities and early childhood 
education programs has brought about a growing concern with respect to the 
type and quality of care provided for children. The question of quality 
of early childhood care is crucial as suggested by research on the effects 
of the preschool environmait and the developnent of intellectual and social- 
©TOtional corpetencies in young children. With an increasing nuitber of 
children receiving care outside the home it is .trrportant to understaivi the 
effects this has on children. Thus, the key issues in research and evaluation 
in early childhood programs are similar to those at any educational level. 
However, as Ryan and Moffitt (1974) suggest traditional models and methods 
of evaluation research are inaj^ropriate if they only eftploy the classical 
experimental model. 

The traditional models that Ryan and Moffitt (1974)^(Moffitt and 
Ryan, 1975) refer to generally are outcone models of evalvjation research. 
Most evaluation of early childhood programs has taken this approach and 
report findings such as "remarkable and significant gains have been made in 
linguistic functioning" (Taylor et al. 1973) as a result of some particular 
program. Any number of other studies report similar findings. Beller (1973) 
reviews 15 studies of early childhood programs and only two mention any type 
of evaluation other than outcome evaluation. Even thai the mention of other 
forms of evaluation was at best cursory. Essentially the traditional psycho- 
metric approach avoids many real issues. For instance, vAiat aspects of the 
teaching procedure or teacher's behavior accounts for any changes that occur? 
If there is a lack of change what aspects of the program may account for it? 
What the differences among various programs? The psychometric approach * 
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offers no solutions to these questions and can not indicate differences 
between programs that result in the varied outcomes reported. For example, 
a study by de Lacey et al. (1973) reports a marked superiority of a 
structured preschool program over the more traditional preschool program. 
However, there is no indication c.s to the processes that account for the 
different results. Most other stixiies can be equally criticized. This is 
not to say that outcomes are uninportant. It is necessary in seme cases to 
delineate programs that are the most effective for increasing outcomes 
such as language and intellectual skills in children. Hdwever, differences 
in programs can only be determined through process evaluation. A good 
exairple of this is provided by Charters and Jones (1973) . They refer to a 
study v*uch was designed to assess the outcome effects of differentiated 
staffing on student achievenent. The evaluation sttidy reached the conclusion 
of no consistent difference in achievement gain^ , 'he dependent variable, 
between differentiated and conventional staffing. Cfiarters and Jones, who 
were at the same time undertaking a process revaluation, clearly show that 
there were no differences in the independent variable, the type of staffing, 
either. Thus, the programs v*uch were evaluated and prodix:ed similar outcomes 
were not in fact different and should not have been expected to produce 
different results. As Charters and Jones point out, the outcome evaluation 
evaluated a non-event. This example indicates clearly the necessity of 
substantially more corprehensive research to analyze the content and processes 
of the programs to enable an understanding oi the outcomes. 

One method of broader.ing the research in early childhood education 
is to assume a medical model as suggested by Messick and Barrows (1972) . 
As they indicate the medical model recognizes that prescriptions for treatment 
and its effectiveness must take into account not only the symptoms reported 
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but also the character of the orgaiusm, its ecology, and the processes 
that produce the desired effects. The r^naining portion of the paper will 
exandne process research as xt relates to early childhood programs in three 
main categories, type of instrurrentation, purposes of instruments and 
methodological considerations. 

There are many types of instruments available tlBt teve been adapted 
for use in observing interaction betweei teachers and children. In fact, 
Rosenshine and Furst (1973) indicate that they have documented 120 different 
systems of process observation and this they see as an underestimate. With 
this number of observational techniques available it is necessary to categorize 
the various styles of the instnaments. Gordon and Jester (1373) suggest a 
series of categories based on previous research. The initial style of 
observational technique is referred to as specimen description, in vAiich 
ti:e sequences of interaction are recorded by continuous rK)te taking or audio-* 
video tape. This style is characterized by a detailed recording of the 
behavior observed and is referred to as an open or low iriference system as | 
the raw data is available even after the observation is cotpleted^ This 
contrasts to a closed or high inference system of observation in vAiich the 
data is coded during the observation. As such^the actual betevior of the 
individuals is not available - rather behaviors may be noted on a check list 
as they occur and inferences must then be nede regarding the actual teaching 
behavior. There are obvious advantages to both systems. The open system 
offers a detailed record of behavior that can be examined with no inferaices 
being made about the teaching behavior, analyzed and coded in a nuntoer of 
different ways over time. However, it represents a relatively expensive 
way of gathering data in time taken to collect, axicilyze and code the data. 
Closed systems allow for a sanpling of behavior in a more efficient nenner^ 
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However, probleans of interrater reliability, the fact that the actual 
behavioral interactions are not available and inferences nust be made 
regarding the data, represent dravAxicks to the closed systems. Ideally 
the open system using video tape reixoaents the most canprehensive method 
of examining prxx:ess variables. :lowever, in many cases due to financial or 
time constrictions researchers must be satisfied with less than the ideal 
situation and work within the structure of a closed system. A type of 
closed system which has been used relatively extensively is referred to by 
Gordon and Jester (1973) as Time Sarrpling which is divided into sign and 
category systems. As Gordon and Jester (1973) indicate sign systems retain 
some elonents of the specific behavior observed as they are listed in the 
observer's record form and he checks them off as they occur vAiereas in 
category systems this record of behavior is not available as the behavior 
exhibited is grov^ied under various labels. 

Two other types of closed systems referred to by Gordon and Jester 
(1973) are event sanpling and trait rating. During event sampling the 
observer records the various events that occur, such as argunents, as opposed 
to noting the specific behavior of individuals. In trait rating the observer 
notes the behavior and then makes an inference regarding the trait possessed 
by the person being observed. This latter metiiod is, to say the least, 
somewhat subjective. If a closed system is to be utilized, a combination 
of scales which provide a time/sign and event sanpling probably provide the 
nost useful data for prc>cess ctnalysis. 

Process analysis can serve a number of different purposes. Itosen- 
shine and Furst (1973) suggest four useful classifications for process 
analysis. The first is to describe current teaching practices. As is 
obvious, description is the predominent pirpose and the data from this type 
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of analysis is intended to provide son>e basic data on teaching in early 
childhood settings. In other wrds, it provides information regarding the 
state of early childhood programs. Once there is an adequate baseline, 
these descriptive stidies shDuld lead to correlational and experinental 
studies rather than more descriptive studies. However, in relation to the 
schcol system experimental studies have not followed descriptive studies, 
rather there has been a plethora of descriptive studies. It is to be hoped 
that research in early childhood education will not follow this pattern. 

The second purpose process analysis can be utilized for is to trair\ 
teachers (RosenshJjie and Furst, 1973) . Process analysis can provide a 
teacher, experienced or inexperienced, with feedback on his own behavior. 
As well, it can provide teachers with procedures to classify instructional 
methods and models of instaK:tion. This function could be especially useful 
in training teachers for early childhood programs. For instarK:e, if a 
specimen description method through the use of video tape was used to observe 
beginning teachers and the beginning teachr analyzed the teaching process 
with the aid of an expert teacher it is likely that new teachers would gain 
considerable insight into the teaching process. Further, if video tapes 
of an expert teacher workir^ with a class wer.. available, the new teacher 
would be able to contrast and compare teachiiig styles and attetpt to model 
some of the behavior to discover which teaching methods or styles would be 
effective for him. 

A third function of process analysis is to monitor instructional 
programs (Rosenshine and Furst, 1973) . There are many experimental approaches 
being inplemented in early childhood education to discover tne most effective 
ways of working with young children. Any two programs may have quite 
different goals yet the question can still be raised as to whether the actual 
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functioning of the programs is different. The exanple provided by 
Charters ard Jones (1973) and discussed previously indicates the necessity 
of monitoring progrcims to discover if the intiintions of the program 
developer are being iitplementad. Without evidence that the processes 

•.V 

involved in various programs differ, educators should not assume that 
different goals necessarily mean different instructional behavior within 
prograins. If in practice the instructional activities in a particular 
program represent low iirplementation of a program, it is xinreasonable to 
judge the success or failure of the program as it has not ixen inpleraented 
in any true sense. 

The final function of process evaluation is to investigate relation- 
ships between classroan activities and piipil growth (Rosenshine and Furst/ 
1973) . This function is particularly iitportant for early childhood programs 
aixa represents a combination of outcome and process analysis. The outoanes 
of a particular program are measured perhaps through a pre-test, post- test 
methodology and are then related to the teaching process through pro«5ss 
evaluation. One instnament which has been used in this fashion is this 
Observation Schedule and Record (OSCAR) (Medley and ftetzel, 1958) . However, 
the OSCAR needs sane revision to be used in the preschool setting. S<2veral 
recent studies (Moffitt and \975; Taylor, 1975) have successfully 

used the OSCAR to evaluate programs and relate outcaiies and processes- Sane 
earlier work by Soar and Soar (1972) utilizing tl-ie Teacher Practicos 
Observation Record, has also related outcones and processes. However, on 
the whole nost studies examine either outcomes or processes in isolation. 
At the presCTt stage, the research in early childhood education should be 
noving away from these sircar evaluations. Yet many persons working in 
the area neglect the process analysis. For exartple, a recent paper ty Vane 
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(1976) entitled Problcjns In and Strategies For EvaluatJbng Preschool Programs; 
totally ignores the whole area of process evaluation and ooncentrates only 
on outcome research. As Moffitt and (1975) indicate/ traditioned 

nrdels of evaluation are inappropriate for early cliildhood educatic»i. 
Researchers must begiji to investigate in a detailed and systeratic sense 
the relationship bet^A^een classroom activities and student grovrt±i. 

There are a number ot methodological issues ttet should be con- 
sidered before a process analysis is undertaken. One of the major problenis 
is deciding on the appropriate instrument to observe the process variables. 
There is a large pool of instruments for observing educational settings 
cuxl rather than developing new instruments researchers should begin to 
utilize existing ones to test their usefulness. Many instruments have haen 
utilized in only the study for which they were developed. There is no 
need for a further proliferation of instruments. Soar and Soar (1972) 
suggest several useful instruments/ the Florida Affective Categories Measure 
and Teacher Practices Observation Record. As well, with sane modification, 
the Observation Schedule arx3 Record developed by Medley and Metzel (1958) 
will prove to be a useful tool. These instruments or a combination of 
than will, in all likelihood/ sairple the variables that most researchers 
vash to examine. Thus, the re.sults of various studies will be ccnparable 
as they will examine canroi variables. Further, as many researchers use 
thesp instruments it will provide a broad data base. As Rosenshine and 
Furst (1973) iiidicate, it is best to be suspicious of variables and instruments 
which do not have a data base. 

A second methodological issue relates to observer reliability. 
Typically reliability coefficients are calculated on total talleys. VJhat 
is necessary is to calculate reliability coefficients on each event observed 
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to clitxx)ver if the obaervars are in atjrcanent regardiiig specif ic events 
(Mitchell, 1969), Observer reliability will reach an acceptable level when 
the observers are well trained and yiven a thorouyh explanation regarding 
the specific details of the observational system. Medley and Metzel (1963) 
discuss other issues relating to the reliability of observational tochniqvKis 
and the reader is referred to their paper. 

A final issue, though not strictly methodological, relates to 
tilt; degree of control the researcher has over selection and supervision of 
the progrcim being evaluated. In the experimental preschool the researcher 
may have considerable control. However, these controlled programs may not 
be typical of the nature of early childhood programs. Evaluations of 
programs other than in demonstration settings is clearly needed as many 
times programs develoEsed through demonstration projects are difficult to 
inplement in the field. In field settings the researcher may have little 
control. However, the ideal situation is one in which the evaluator and 
sponsors of the program work in a cooperative effort. It has been the 
author's experience that when sponsors are made aware of the necessity of 
evaluation and clearly understand the feedback they will receive, cooperation 
is generally forthcoming. 

In conclusion^ it is fair to say that only through both outccrne 
arid process evaluation will it be possible to develop procedures on how to 
best educate young children. Evalijations which take into acccxant the issues 
raised in the presen'.- paper 'All sho-^ where there needs to be further res^ch 
in early childhood educ ition and suggest id^s for theories starting from 
eipirical data as well as testing already established theories. As yet 
process analysis of the preschool situation is still just in the beginning 
stages and there is the need to carry out conprehensive investigations of 
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early childhood iircxinutu to iiicrcviiKs our kncwlodgo of tho rvituro of pro- 
acVwol teaching ««3 Iciming arri the oCfects preschool prograna hava on 
young children. 
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